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CAPSULE  SUMMARY:  Benefits  from  common  modeling  infrastructure  and  component 
interface  standards  are  being  realized  in  a  suite  of  national  weather  and  climate  codes. 

ABSTRACT 

The  Earth  System  Prediction  Suite  (ESPS)  is  a  collection  of  flagship  U.S.  weather  and  climate 
models  and  model  components  that  are  being  instrumented  to  conform  to  interoperability 
conventions,  documented  to  follow  metadata  standards,  and  made  available  either  under  open 
source  terms  or  to  credentialed  users. 

The  ESPS  represents  a  culmination  of  efforts  to  create  a  common  Earth  system  model 
architecture,  and  the  advent  of  increasingly  coordinated  model  development  activities  in  the  U.S. 
ESPS  component  interfaces  are  based  on  the  Earth  System  Modeling  Framework  (ESMF), 
community-developed  software  for  building  and  coupling  models,  and  the  National  Unified 
Operational  Prediction  Capability  (NUOPC)  Layer,  a  set  of  ESMF-based  component  templates 
and  interoperability  conventions.  This  shared  infrastructure  simplifies  the  process  of  model 
coupling  by  guaranteeing  that  components  conform  to  a  set  of  technical  and  semantic  behaviors. 
The  ESPS  encourages  distributed,  multi-agency  development  of  modeling  systems,  controlled 
experimentation  and  testing,  and  exploration  of  novel  model  configurations,  such  as  those 
motivated  by  research  involving  managed  and  interactive  ensembles.  ESPS  codes  include  the 
Navy  Global  Environmental  Model  (NavGEM),  HYbrid  Coordinate  Ocean  Model  (HYCOM), 
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and  Coupled  Ocean  Atmosphere  Mesoscale  Prediction  System  (COAMPS®);  the  NOAA 
Environmental  Modeling  System  (NEMS)  and  the  Modular  Ocean  Model  (MOM);  the 
Community  Earth  System  Model  (CESM);  and  the  NASA  ModelE  climate  model  and  GEOS-5 
atmospheric  general  circulation  model. 

BODY  TEXT 

The  software  infrastructure  that  underlies  Earth  system  models  includes  workhorse  utilities  as 
well  as  libraries  generated  by  research  efforts  in  computer  science,  mathematics,  and 
computational  physics.  The  utilities  cover  tasks  like  time  management  and  error  handling,  while 
research-driven  libraries  include  areas  such  as  high  performance  I/O,  algorithms  for  grid 
remapping,  and  programming  tools  for  optimizing  software  on  emerging  computer  architectures. 
Collectively,  this  model  infrastructure  represents  a  significant  investment.  As  a  crude 
comparison,  a  comprehensive  infrastructure  package  like  the  Earth  System  Modeling  Framework 
(ESMF;  Hill  et  al.  2004,  Collins  et  al.  2005),  is  comparable  in  size  to  the  Community  Earth 
System  Model  (CESM;  Hurrell  et  al.  2013),  each  at  just  under  a  million  lines  of  code.1 

In  2002,  Dickinson  et  al.  articulated  the  goal  of  common  model  infrastructure,  a  code  base  that 
multiple  weather  and  climate  modeling  centers  could  share.  This  idea  was  shaped  by  an  ad-hoc, 
multi-agency  working  group  that  had  started  meeting  several  years  earlier,  and  was  echoed  in 
reports  on  the  state  of  U.S.  climate  modeling  (NRC  1998,  NRC  2001,  Rood  et  al.  2000).  Leads 
from  research  and  operational  centers  posited  that  common  infrastructure  had  the  potential  to 
foster  collaborative  development  and  transfer  of  knowledge;  lessen  redundant  code;  advance 


1  Codes  compared  are  CESM  1.0.3,  at  about  820K  lines  of  code  (Alexander  and  Easterbrook  201 1),  and  ESMF 
6.3.0rpl,  at  about  920K  lines  of  code  (ESMF  metrics  available  online  at: 
https://www.earthsystemcog.org/proiects/esmf/sloc_annual) 
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65  computational  capabilities,  model  performance  and  predictive  skill;  and  enable  controlled 

66  experimentation  in  coupled  systems  and  ensembles.  This  vision  of  shared  infrastructure  has  been 

67  revisited  in  more  recent  publications  and  venues;  for  example,  in  the  2012  National  Research 

68  Council  report  entitled  A  National  Strategy  for  Advancing  Climate  Modeling  (NRC  2012). 

69  In  this  article  we  describe  how  the  vision  of  common  infrastructure  is  being  realized,  and  how  it 

70  is  changing  the  approach  to  Earth  system  modeling  in  the  U.S.  Central  to  its  implementation  is 

71  an  Earth  System  Prediction  Suite  (ESPS),  a  collection  of  weather  and  climate  models  and  model 

72  components  that  are  being  instrumented  to  conform  to  interoperability  conventions,  documented 

73  to  follow  metadata  standards,  and  made  available  either  under  open  source  terms  or  to 

74  credentialed  users. 

75  We  begin  by  discussing  how  the  U.S.  modeling  community  has  evolved  toward  a  common 

76  architecture,  and  explain  the  role  of  the  ESMF  and  related  projects  in  translating  that 

77  convergence  into  technical  interoperability.  We  define  what  we  mean  by  minimal 

78  interoperability  and  the  behavioral  rules  needed  to  achieve  it,  and  describe  the  ESPS  code  suite 

79  and  its  target  inclusion  criteria.  We  give  examples  of  the  adoption  process  for  different  kinds  of 

80  codes,  and  of  science  enabled  by  common  infrastructure.  Finally,  we  examine  the  potential  role 

81  of  the  ESPS  in  model  ensembles,  and  consider  areas  for  future  work. 

82  EMERGENCE  OF  A  COMMON  MODEL  ARCHITECTURE 

83  Several  generations  of  model  infrastructure  development,  described  in  the  sidebar  (Linked  and 

84  Leveraged  . . .)  allowed  for  the  evolution  and  evaluation  of  design  strategies.  A  community  of 

85  infrastructure  developers  emerged,  whose  members  exchanged  ideas  through  a  series  of 

86  international  meetings  focused  on  coupling  techniques  (e.g.  Dunlap  et  al.  2014),  comparative 
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analyses  such  as  Valcke  et  al.  (2012),  and  design  reviews  and  working  group  discussions  hosted 
by  community  projects  such  as  CESM  and  ESMF. 

Over  time,  model  developers  from  major  U.S.  centers  implemented  similar  model  coupling 
approaches,  based  on  a  small  set  of  frameworks:  ESMF,  the  CESM  Coupler  7  (CESM  CPL7; 
Craig  et  al.  2012),  which  uses  the  lower-level  Model  Coupling  Toolkit  for  many  operations 
(MCT;  Larson  et  al.  2005,  Jacob  et  al.  2005),  and  the  Flexible  Modeling  System  (FMS;  Balaji 

2012) .  ESMF,  CPL7,  and  FMS  share  several  key  architectural  characteristics.  First,  they  are  all 
single  executable  frameworks,  meaning  that  constituent  components  are  called  as  subroutines  by 
a  top-level  driver.  Second,  major  physical  domains  such  as  atmosphere,  ocean,  land,  sea  ice,  and 
wave  models  are  wrapped  with  component  interfaces,  and  the  component  interfaces  are 
structured  similarly,  with  arguments  for  fields  imported,  fields  exported,  and  time  information. 
Not  all  coupling  technologies  follow  these  patterns.  For  example,  in  the  OASIS  coupler  (Valcke 

2013)  used  by  many  European  climate  models,  components  are  run  as  separate,  linked  programs 
or  “multiple  executables”  and  in  general  do  not  require  that  fields  transferred  between 
components  pass  through  a  component  interface. 

The  design  convergence  of  U.S.  models  created  an  opportunity  for  coordination  that  a  new 
program  was  ready  to  exploit.  The  National  Unified  Operational  Prediction  Capability  (NUOPC; 
see  http://www.nws.noaa.gov/nuopc/),  a  consortium  of  operational  weather  prediction  centers 
and  their  research  partners,  was  established  in  2007  with  goals  that  included  creating  a  global 
atmospheric  ensemble  weather  prediction  system  and  promoting  collaborative  model 
development.  In  support  of  these  goals,  NUOPC  sought  further  standardization  of  model 
infrastructure,  and  formalized  the  concept  of  common  model  architecture  (CMA;  Sandgathe  et 
al.  2009;  McCarren  et  al.  2013).  The  CMA  can  be  defined  as  a  set  of  conventions  that  govern 
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the  application  programming  interfaces  (APIs)  of  model  components,  the  “level  of 
componentization,”  and  the  protocols  for  component  interaction.  In  general  terms,  models  using 
the  ESMF,  CPL7,  or  FMS  frameworks  could  be  said  to  share  the  same  CMA. 

Despite  the  similarities  in  structure,  the  components  under  these  different  frameworks  still 
required  the  implementation  of  a  common  translation  layer  to  achieve  a  minimal  level  of 
interoperability.  NUOPC  defined  this  minimal  level  of  interoperability  as  the  ability  of  a 
component  to  execute  without  code  changes  within  a  driver  that  provides  the  fields  that  it 
requires,  and  to  return  with  informative  messages  if  its  input  requirements  are  not  met.  Unlike 
FMS  and  CESM,  which  are  associated  with  specific  modeling  systems,  the  ESMF  software  is 
intended  to  support  multiple  modeling  systems,  and  it  emerged  as  the  reference  architecture  and 
CMA  implementation.  With  ESMF,  the  NUOPC  consortium  undertook  formal  codification  of 
the  CMA  and  its  realization  in  widely  usable  (e.g.  portable,  reliable,  efficient,  documented) 
software. 

ESMF  AND  THE  NUOPC  LAYER 

ESMF  is  high  performance  software  for  building  and  coupling  Earth  system  models.  It  includes 
a  superstructure  for  representing  model  and  coupler  components  and  an  infrastructure  of 
commonly  used  utilities,  including  grid  remapping,  time  management,  model  documentation, 
and  data  communications  (see  https://www.earthsystemcog.org/projects/esmf/).  It  was 
developed  and  is  governed  by  a  set  of  multi-agency  partners  that  includes  NASA,  NOAA,  the 
Department  of  Defense  and  the  National  Science  Foundation.  ESMF  can  be  used  in  multiple 
ways:  1)  to  create  interoperable  component-based  modeling  systems;  2)  as  a  source  of  libraries 
for  commonly  used  utilities;  3)  as  a  file-based  offline  generator  of  interpolation  weights  for  many 
different  kinds  of  grids;  and  4)  as  a  Python  package  for  grid  remapping. 
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The  ESMF  design,  which  evolved  over  a  period  of  years  through  weekly  community  reviews  and 
thousands  of  user  support  interactions,  accommodates  a  wide  range  of  data  structures,  grids,  and 
component  layout  and  sequencing  options.  The  main  constructs  are  gridded  components 
(ESMF_GridComp)  and  coupler  components  (ESMF_CplComp).  Physical  fields  are 
represented  using  ESMF_Fields,  which  are  contained  in  import  and  export  ESMF_State 
objects  in  order  to  be  passed  between  components.  ESMF  defines  three  standard  methods: 
initialize,  run,  and  finalize,  which  can  have  multiple  phases;  however,  there  are  no  requirements 
on  how  these  methods  should  behave.  Since  ESMF  data  structures  can  often  reference  native 
model  data  structures  and  ESMF  methods  can  invoke  model  methods  without  introducing 
significant  performance  overhead,  the  software  can  serve  either  as  a  primary  infrastructure  or  as 
a  wrapper  around  components  in  existing  coupled  models. 

ESMF  provides  interfaces  and  data  structures  with  few  constraints  about  how  to  use  them.  This 
flexibility  enabled  it  to  be  adopted  by  many  modeling  systems,  but  limited  the  interoperability 
across  these  systems.  To  address  this  issue,  the  NUOPC  consortium  developed  a  set  of  coupling 
conventions  and  generic  representations  of  modeling  system  elements  -  drivers,  models, 
connectors,  and  mediators  -  called  the  NUOPC  Layer  (see 

http://www.earthsystemcog.org/projects/nuopc/).  NUOPC  drivers  and  models  can  be 
understood  in  the  usual  way;  connectors  handle  simple  data  transformations  and  transfers,  and 
mediators  implement  field  merges  and  custom  coupling  code.  Table  1  summarizes  NUOPC 
generic  components  and  their  roles.  In  some  cases,  the  generic  components  may  be  used  without 
modification;  in  others,  user  code  is  added  at  clear  specialization  points.  Calls  to  NUOPC 
methods  mainly  relate  to  component  creation  and  sequencing,  and  may  be  mixed  with  calls  to 

2  ESMF  components  are  listed  here:  https://www.earthsystemcog.org/proiects/esmf/components 
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155  ESMF  time  management,  grid  remapping,  and  other  methods. 

156  The  NUOPC  Layer  enables  multi-component  systems,  including  hierarchies  and  ensembles,  to 

157  be  assembled  using  pre-fabricated  code.  Figure  1  is  a  schematic  of  two  simple  model 

158  configurations  built  using  generic  components. 

159  While  use  of  the  NUOPC  Layer  cannot  guarantee  scientific  compatibility,  it  does  guarantee  a  set 

160  of  component  behaviors  related  to  technical  interoperability.  These  are  described  in  the  NUOPC 

161  Layer  Reference  (2014).  Specifically,  it  ensures  that  a  component  will  provide: 

162  (i)  A  GNU  makefile  fragment  that  defines  a  small  set  of  prescribed  variables,  which  a  NUOPC 

163  application  uses  to  compile  and  link  with  the  component. 

164  (ii)  A  single  public  entry  point,  called  SetServices.  Standardizing  this  name  enables  code  that 

165  registers  components  to  be  written  generically. 

166  (iii)  An  InitializePhaseMap,  which  describes  a  sequence  of  standard  initialize  phases  drawn 

167  from  a  set  of  Initialize  Phase  Definitions.  For  example,  one  standard  phase  advertises  the 

168  fields  a  component  can  provide,  based  on  standard  names  drawn  from  the  Climate  and 

169  Forecast  conventions  (CF;  Eaton  et  al.  201 1).  Field  names  are  checked  and  mapped  to  each 

170  other  using  a  NUOPC  Field  Dictionary.  Another  standard  phase  instantiates  the  fields  that 

171  will  be  used. 

172  (iv)A  RunPhaseMap,  in  which  each  phase  must  check  the  incoming  clock  of  the  driver  and  the 

173  timestamps  of  incoming  fields  against  its  own  clock  for  compatibility.  The  component 

174  returns  an  error  if  incompatibilities  are  detected. 

175  (v)  Time  stamps  on  its  exported  fields  consistent  with  the  internal  clock  of  the  component. 

176  (vi)  A  finalize  method  that  cleans  up  all  allocations  and  file  handles. 
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These  constraints,  involving  build  dependencies,  initialization  sequencing,  and  run  sequencing, 
are  the  focus  of  the  NUOPC  Layer  because  they  are  required  to  satisfy  the  definition  of  minimal 
interoperability:  that  components  will  run  without  code  changes  if  their  required  field  inputs  are 
satisfied,  and  will  return  with  appropriate  warnings  if  they  are  not.  The  constraints  nonetheless 
allow  for  the  representation  of  many  different  model  control  sequences.  They  also  enable 
negotiation  and  contingencies  to  be  represented  in  a  structured  way,  a  feature  that  becomes 
important  in  optimization  of  multi-component  systems,  where  components  may  compete  for 
resources. 

The  ESMF/NUOPC  software  distribution  is  suitable  for  broad  use  as  it  has  an  open  source 
license,  comprehensive  user  documentation,  a  suite  of  about  6500  regression  tests  that  runs 
nightly  on  about  30  different  platform/compiler  combinations,  and  a  user  support  team. 
Performance  evaluation  occurs  on  an  ongoing  basis,  with  reports  posted  at 
https://www.earthsystemcog.org/projects/esmf/performance.  The  software  has  about  6000 
registered  downloads. 

THE  EARTH  SYSTEM  PREDICTION  SUITE 

The  National  Earth  System  Prediction  Capability  (National  ESPC;  see  http://espc.oar.noaa.gov) 
combines  the  ESPC,  initiated  in  2010,  and  NUOPC,  to  extend  the  scope  of  the  NUOPC  program 
in  several  ways.  The  National  ESPC  goal  is  a  global  Earth  system  analysis  and  prediction 
system  that  will  provide  seamless  predictions  from  days  to  decades,  developed  with 
contributions  from  a  broad  community.  Expanding  on  NUOPC,  the  National  ESPC  includes 
additional  research  agency  partners  (NSF,  NASA,  and  DOE),  time  scales  of  prediction  that 
extend  beyond  short  term  forecasts,  and  new  modeling  components  (e.g.  cryosphere,  space). 
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In  order  to  realize  the  National  ESPC  vision,  major  U.S.  models  must  be  able  to  share  and 
exchange  model  components.  Thus  the  National  ESPC  project  is  coordinating  development  of  an 
Earth  System  Prediction  Suite  (ESPS),  a  collection  of  NUOPC-compliant  Earth  system 
components  and  model  codes  that  are  technically  interoperable,  tested,  documented,  and 
available  for  integration  and  use.  At  this  stage,  ESPS  focuses  on  coupled  modeling  systems  and 
atmosphere,  ocean,  ice  and  wave  components. 

ESPS  partners  are  targeting  the  following  inclusion  criteria: 

•  ESPS  components  and  coupled  modeling  systems  are  NUOPC-compliant. 

•  ESPS  codes  are  versioned. 

•  Model  documentation  is  provided  for  each  version  of  the  ESPS  component  or 
modeling  system. 

•  ESPS  codes  have  clear  terms  of  use  (e.g.  public  domain  statement,  open  source 
license,  proprietary  status),  and  have  a  way  for  credentialed  ESPC  collaborators  to 
request  access. 

•  Regression  tests  are  provided  for  each  component  and  modeling  system. 

•  There  is  a  commitment  to  continued  NUOPC  compliance  and  ESPS  participation  for 
new  versions  of  the  code. 

ESPS  is  intended  to  formalize  the  steps  in  preparing  codes  for  cross-agency  application,  and 
the  inclusion  criteria  support  this  objective.  NUOPC  compliance  guarantees  a  well-defined, 
minimal  level  of  interoperability,  and  enables  assembly  of  codes  from  multiple  contributors. 
Versioning  is  essential  for  traceability.  Structured  model  documentation  facilitates  model 
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analysis  and  intercomparison.  Clear  terms  of  use  and  a  way  to  request  code  access  are 
fundamental  to  the  exchange  of  codes  across  organizations.  Regression  tests  are  needed  for 
verification  of  correct  operation  on  multiple  computer  platforms.  The  commitment  to 
continued  participation  establishes  ESPS  as  an  ongoing,  evolving  capability. 

At  the  time  of  this  writing,  not  all  criteria  are  satisfied  for  all  candidate  codes.  Further,  the 
criteria  themselves  are  likely  to  evolve.  The  extent  of  the  metadata  to  be  collected  still  needs 
to  be  determined,  and  specific  requirements  for  regression  tests  have  not  yet  been 
established.  The  process  of  refining  the  inclusion  criteria  and  completing  it  for  all  codes  is 
likely  to  occur  over  a  period  of  years.  However,  a  framework  is  now  in  place  for  moving 
forward.  Current  information  is  presented  on  the  ESPS  webpage,  see 
https://www.earthsystemcog.org/projects/esps/. 

Code  Development,  Compliance  Checking,  and  Training  Tools 
The  viability  of  ESPS  depends  on  there  being  a  straightforward  path  to  writing  compliant 
components.  Several  tools  are  available  to  facilitate  development  and  compliance  verification  of 
ESPS  components  and  coupled  models.  These  include  the  command  line-based  NUOPC 
Compliance  Checker  and  Component  Explorer,  both  described  in  the  NUOPC  Layer  Reference 
(2014),  and  the  graphical  Cupid  Integrated  Development  Environment  (IDE)  (Dunlap  2014). 

The  NUOPC  Compliance  Checker  is  an  analysis  tool  that  intercepts  component  actions  during 
the  execution  of  a  modeling  application  and  assesses  whether  they  conform  to  standard  NUOPC 
Layer  behaviors.  It  is  linked  by  default  to  every  application  that  uses  ESMF  and  can  be  activated 
by  setting  an  environment  variable.  When  deactivated,  it  imposes  no  performance  penalty.  The 

3  Initial,  minimal  metadata  associated  with  each  ESPS  model  is  being  collected  and  displayed  using  tools  from  the 
Earth  System  Documentation  consortium  (ES-DOC;  Lawrence  et  al.  2012). 
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241  Compliance  Checker  produces  a  compliance  report  that  includes,  for  each  component  in  an 

242  application,  information  such  as  checks  for  presence  of  the  required  initialize,  run,  and  finalize 

243  phases,  correct  timekeeping,  how  fields  are  passed  between  components,  and  the  presence  of 

244  required  component  and  field  metadata. 

245  The  Component  Explorer  is  a  run-time  tool  that  analyzes  a  single  model  component  by  acting  as 

246  its  driver.  The  tool  offers  a  way  of  evaluating  the  behavior  of  the  component  outside  of  a  coupled 

247  modeling  application.  It  steps  systematically  through  the  phases  defined  by  the  component  and 

248  performs  checks  such  as  whether  required  makefile  fragments  are  provided,  whether  a  NUOPC 

249  driver  can  link  to  the  component,  and  whether  error  messages  are  generated  if  the  required  inputs 

250  are  not  supplied.  For  additional  information,  the  Compliance  Checker  can  be  turned  on  while  the 

25 1  Component  Explorer  is  running.  A  test  of  NUOPC  compliance  is  running  the  candidate 

252  component  in  the  Component  Explorer  and  ensuring  that  it  generates  no  warnings  from  the 

253  Compliance  Checker  when  it  is  turned  on. 

254  Cupid  provides  a  comprehensive  code  editing,  compilation,  and  execution  environment  with 

255  specialized  capabilities  for  working  with  NUOPC-based  codes.  It  is  implemented  as  a  plugin  for 

256  Eclipse,  a  widely  used  IDE.  A  key  feature  of  Cupid  is  the  ability  to  create  an  outline  that  shows 

257  the  NUOPC-wrapped  components  in  the  application,  their  initialize,  run,  and  finalize  phases,  and 

258  their  compliance  status.  The  outline  is  presented  to  the  developer  side-by-side  with  a  code  editor, 

259  and  a  command  line  interface  for  compiling  and  running  jobs.  Cupid  provides  contextual 

260  guidance  and  can  automatically  generate  portions  of  the  code  needed  for  compliance.  The  user 

261  can  select  several  prototype  codes  for  training,  or  can  import  their  own  model  code  into  the 

262  environment.  Figure  2  shows  the  Cupid  graphical  user  interface. 
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263  Table  3  summarizes  the  tools  described  in  this  section  and  their  main  uses.  Static  analysis  mode 

264  refers  to  the  examination  of  code,  while  dynamic  analysis  mode  refers  to  evaluation  of 

265  component  behaviors  during  run-time. 

266  ADAPTING  MODELS  FOR  ESPS 

267  In  this  section,  we  describe  the  approach  to  adapting  different  sorts  of  codes  for  ESPS.  We  look 

268  at  implementation  of  single  model  components,  wholly  new  coupled  systems,  and  existing 

269  coupled  systems. 

270  The  realities  of  implementation  required  adjustments  to  some  goals  and  strategies.  Most 

271  significantly,  the  idea  that  a  single  common  software  framework  must  replace  all  others,  a 

272  solution  advanced  in  the  2012  NRC  report,  proved  unrealistic  and  unnecessary.  In  practice,  it 

273  has  been  more  effective  to  wrap  and  combine  multiple  infrastructure  packages,  and  ESMF  often 

274  co-exists  with  native  infrastructure  within  modeling  applications.  This  approach  also  enables 

275  centers  to  maintain  local  differences  in  coupling  methodologies;  longstanding  coupled  modeling 

276  efforts  at  NCAR,  GFDL,  and  NASA  have  established  organizational  preferences  for  handling 

277  coastlines,  conservative  transfer  of  fluxes,  and  other  coupling  operations.  The  details  of  these 

278  operations  are  not  reviewed  here;  detailed  discussion  of  techniques  is  available  in  documents 

279  such  as  Craig  (2014).  The  different  approaches  encountered  to  date  can  be  accommodated  by  the 

280  NUOPC  Layer  rules  and  software. 

281  Single  model  components  are  the  most  straightforward  to  wrap  with  NUOPC  Layer  interfaces. 

282  The  Modular  Ocean  Model  (MOM5;  Griffies  2012)  and  Hybrid  Coordinate  Ocean  Model 

283  (HYCOM;  Halliwell  et  al.,  1998,  Halliwell  et  al.,  2000,  Bleck,  2002)  are  examples  of  this  case. 

284  Both  ocean  models  had  previously  been  wrapped  with  ESMF  interfaces,  and  had  the  distinct 
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285  initialize,  run,  and  finalize  standard  methods  required  by  the  framework.  For  NUOPC 

286  compliance,  a  standard  sequence  of  initialize  phases  was  added,  and  conformance  with  the  Field 

287  Dictionary  checked.  The  process  of  wrapping  MOM5  and  HYCOM  with  NUOPC  Layer  code 

288  required  minimal  changes  to  the  existing  model  infrastructure.  For  both  MOM5  and  HYCOM, 

289  NUOPC  changes  can  be  switched  off,  and  MOM5  can  still  run  with  GFDL’s  in-house  FMS 

290  framework. 

291  The  construction  of  newly  coupled  systems  is  a  next  step  in  complexity.  The  Navy  global 

292  modeling  system  and  the  NOAA  Environmental  Modeling  System  (NEMS;  Iredell  et  al.  2014) 

293  are  examples  in  this  category.  Navy  developers  coupled  the  Navy  Operational  Global 

294  Atmospheric  Prediction  System  (NOGAPs;  Rosmond  1992,  Bayler  and  Lewit  1992)  and 

295  HYCOM  by  introducing  simple  NUOPC  connectors  between  the  models,  and  were  able  to  easily 

296  switch  in  the  newer  Navy  Global  Environmental  Model  atmosphere  (NavGEM;  Hogan  et  al. 

297  2014)  when  it  became  available.  This  work  leveraged  ESMF  component  interfaces  introduced 

298  into  NOGAPS  as  part  of  the  Battlespace  Environments  Institute  (BEI;  Campbell  et  al.  2010).  The 

299  NUOPC-based  HYCOM  code  from  this  coupled  system  was  a  useful  starting  point  for  coupling 

300  HYCOM  with  components  in  NEMS  and  the  CESM. 

301  NEMS  is  an  ambitious  effort  to  organize  a  growing  set  of  operational  models  at  the  National 

302  Centers  for  Environmental  Prediction  under  a  unifying  framework.  Model  coupling  within 

303  NEMS  began  with  coupling  the  Global  Spectral  Model  or  GSM  (previously  the  Global  Forecast 

304  System  or  GFS;  EMC  2003)  to  HYCOM  and  MOM5  ocean  components  and  the  CICE  sea  ice 

305  model  (Hunke  and  Lipscomb  2008).  A  NUOPC  mediator  and  connectors  were  introduced  in 

306  order  to  transfer  and  transform  data  on  a  potentially  different  grid  and  distribution  than  the 

307  component  models,  and  to  perform  merging  and  other  coupling  operations.  A  prototype  of  the 
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308  atmosphere-ocean-ice  system  has  been  completed,  but  much  work  remains  to  validate  the  code, 

309  introduce  additional  components,  and  ready  the  system  for  operational  use.  Other  components 

310  now  being  introduced  into  NEMS  include  the  WaveWatch  3  model  (Tolman  2002),  the 

311  Ionosphere-Plasmasphere  Electrodynamics  (IPE)  model  (based  on  an  earlier  model  described  in 

312  Fuller-Rowell  et  al.  1996  and  Millward  et  al.  1996),  and  a  hydraulic  component  implemented 

313  using  the  WRF-Hydro  model  (Gochis  et  al.  2013).  The  Non-Hydrostatic  Mesoscale  Model 

314  (NMMB;  Janjic  et  al.  2012)  will  be  coupled  within  NEMS  to  the  Princeton  Ocean  Model  (POM; 

315  Blumberg  and  Mellor  1987)  regional  ocean  for  hurricane  forecasts,  and  there  are  also  plans  to 

316  introduce  an  alternate  ice  model,  KISS  (Grumbine  2013).  Shown  schematically  in  Figure  3,  all 

317  are  being  constructed  as  NUOPC  components. 

318  Adapting  an  existing  coupled  modeling  system  for  NUOPC  compliance  is  most  challenging, 

319  since  adoption  must  work  around  the  native  code.  The  CESM,  the  Coupled  Ocean  Atmosphere 

320  Mesoscale  Prediction  System  (COAMPS;  Hodur  1997,  Chen  et  al.  2003),  and  ModelE  (Schmidt 

321  et  al.  2006)  are  examples  of  this.  In  CESM,  a  fully  coupled  model  that  includes  atmosphere, 

322  ocean,  sea  ice,  land  ice,  land,  river  and  wave  components,  ESMF  interfaces  have  been  supported 

323  at  the  component  level  since  2010,  when  it  was  known  as  the  Community  Climate  System  Model 

324  4.0.  However,  the  CESM  driver  was  based  on  the  MCT  data  type.  Recently,  the  driver  was 

325  rewritten  to  accommodate  the  NUOPC  Layer.  By  introducing  a  new  component  data  type  in  the 

326  driver,  either  NUOPC  component  interfaces  or  the  original  component  interfaces  that  use  MCT 

327  data  types  can  be  invoked.  These  changes  did  not  require  significant  modifications  to  the 

328  internals  of  the  model  components  themselves. 

329  Incorporating  the  NUOPC  Layer  into  COAMPS  involved  refactoring  the  existing  ESMF  layer  in 

330  each  of  its  constituent  model  components  and  implementing  a  new  top-level  driver/coupler  layer. 
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331  As  with  the  global  Navy  system,  ESMF  component  interfaces  had  been  introduced  as  part  of 

332  BEI.  The  COAMPS  system  includes  the  non-hydrostatic  COAMPS  atmosphere  model  coupled 

333  to  the  Navy  Coastal  Ocean  Model  (NCOM;  Martin  et  al.  2009)  and  the  Simulating  WAves 

334  Nearshore  model  (SWAN;  Booij  et  al.  1999).  Refactoring  to  introduce  the  NUOPC  Layer  into 

335  each  model  component  involved  changing  the  model  ESMF  initialize  method  into  multiple 

336  standard  phases.  The  representation  of  import/export  fields  was  also  changed  to  use  the  NUOPC 

337  Field  Dictionary.  These  changes  were  straightforward  and  limited  to  the  model  ESMF  wrapper 

338  layer.  An  effort  that  is  just  beginning  involves  wrapping  the  NEPTUNE  [Navy  Environmental 

339  Prediction  system  Utilizing  the  NUMA  (Nonhydrostatic  Unified  Atmospheric  Model)  CorE] 

340  atmosphere,  a  non-hydrostatic  model  which  uses  an  adaptive  grid  scheme  (Kelly  and  Giraldo 

341  2012,  Kopera  et  al.  2014,  Giraldo  et  al.  2013),  with  a  NUOPC  Layer  interface,  as  a  candidate  for 

342  the  Navy’s  next- generation  regional  and  global  prediction  systems.. 

343  When  NUOPC  Layer  implementation  began  in  ModelE,  the  degree  of  coarse-grained 

344  modularization  was  sufficiently  complete  that  the  ModelE  atmosphere  could  be  run  with  four 

345  different  ocean  models  (data,  mixed-layer,  and  two  dynamic  versions),  and  the  two  dynamic 

346  oceans  could  both  be  run  with  a  data  atmosphere.  At  this  time,  atmosphere  and  mixed  layer 

347  ocean  models  are  wrapped  as  NUOPC  components,  and  can  be  driven  using  a  NUOPC  driver. 

348  Specification  of  the  multi-phase  coupled  run  sequence  was  easily  handled  via  NUOPC 

349  constructs.  Mediators  will  provide  crucial  flexibility  to  apply  nontrivial  field  transformations  as 

350  more  complex  coupled  configurations  are  migrated. 

351  Developers  of  the  GEOS-5  atmospheric  model  (Molod  et  al.  2012)  incorporated  ESMF  into  the 

352  model  design  from  the  start,  using  the  framework  to  wrap  both  major  components  and  many  sub- 

353  processes.  In  order  to  fill  in  gaps  in  ESMF  functionality,  the  GEOS-5  development  team 
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354  developed  software  called  the  Modeling  Analysis  and  Prediction  Layer,  or  MAPL.  A  challenge 

355  for  bringing  GEOS-5  into  ESPS  is  translating  the  MAPL  rules  for  components  into  NUOPC 

356  components,  and  vice  versa.  A  joint  analysis  by  leads  from  the  MAPL  and  NUOPC  groups 

357  revealed  that  the  systems  are  fundamentally  similar  in  structure  and  capabilities  (da  Silva  et  al. 

358  2013).  The  feature  that  most  contributes  to  this  compatibility  is  that  neither  NUOPC  nor  MAPL 

359  introduces  new  component  data  types  -  both  are  based  on  components  that  are  native  ESML  data 

360  types  (ESMF_GridComp  and  ESMF_CplComp).  MAPL  has  been  integrated  into  the 

361  ESML/NUOPC  software  distribution,  and  set  up  so  that  refactoring  can  reduce  redundant  code  in 

362  the  two  packages.  Although  the  GEOS-5  model  is  advanced  with  respect  to  its  adoption  of 

363  ESML,  most  of  the  work  in  translating  between  MAPL  and  NUOPC  still  lies  ahead. 

364  RESEARCH  AND  PREDICTION  WITH  COMMUNITY  INFRASTRUCTURE 

365  Community-developed  ESML  and  NUOPC  Layer  infrastructure  supports  scientific  research  and 

366  operational  forecasting.  This  section  describes  examples  of  scientific  advances  that  ESPS  and 

367  related  infrastructure  have  facilitated  at  individual  modeling  centers,  and  the  opportunities  they 

368  bring  to  the  management  of  multi-model  ensembles. 

369  Modeling  and  Data  Center  Impacts 

370  The  use  of  ESML  and  NUOPC  infrastructure  at  modeling  and  data  centers  follows  several 

371  patterns.  The  NUOPC  Layer  allows  software  components  representing  major  physical  realms  to 

372  be  leveraged  across  agencies;  the  underlying  ESMF  architecture  wraps  and  organizes  a  diversity 

373  of  components,  both  large  and  small;  and  ESMF  grid  remapping  and  other  libraries  are  used 

374  extensively  with  coupled  modeling  systems  and  in  other  contexts  such  as  data  visualization. 

375  •  Navy  NavGEM-HYCOM-CICE:  The  NavGEM-HYCOM-CICE  modeling  system,  coupled 
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using  NUOPC  Layer  infrastructure,  is  being  used  for  research  at  the  Naval  Research 
Laboratory.  An  initial  study,  using  just  NavGEM  and  HYCOM,  examined  the  onset  of  a 
Madden  Julien  Oscillation  (MJO)  event  in  201 1  (Peng,  201 1).  For  standalone  NavGEM, 
the  onset  signature  was  basically  absent.  The  coupled  system  was  able  to  reasonably 
simulate  the  onset  signature  compared  with  TRMM  (Tropical  Rainfall  Measuring 
Mission)  measurements.  With  the  addition  of  the  CICE  ice  model,  this  system  is  now 
being  used  to  explore  the  growing  and  melting  of  sea  ice  over  the  Antarctic  and  Arctic 
regions. 

•  COAMPS  and  COAMPS-TC:  The  COAMPS  model  is  run  in  research  and  operations  by  the 
Defense  Department  and  others  for  short-term  numerical  weather  prediction.  COAMPS- 
TC  is  a  configuration  of  COAMPS  specifically  designed  to  improve  tropical  cyclone 
(TC)  forecasts  (Doyle  et  al.  2014).  Both  use  ESMF  and  NUOPC  software  for  component 
coupling.  The  coupled  aspects  of  CO  AMPS  and  COAMPS-TC  were  recently  evaluated 
using  a  comprehensive  observational  data  set  for  Hurricane  Ivan  (Smith  et  al.  2013). 

This  activity  allowed  for  the  evaluation  of  model  performance  based  on  recent 
improvements  to  the  atmospheric,  oceanic,  and  wave  physics,  while  gaining  a  general  but 
improved  understanding  of  the  primary  effects  of  ocean-wave  model  coupling  in  high- 
wind  conditions.  The  new  wind  input  and  dissipation  source  terms  (Babanin  et  al.  2010; 
Rogers  et  al.  2012)  and  wave  drag  coefficient  formulation  (Hwang,  201 1),  based  on  field 
observations,  significantly  improved  SWAN’s  wave  forecasts  for  the  simulations  of 
Hurricane  Ivan  conducted  in  this  study.  In  addition,  the  passing  of  ocean  current 
information  from  NCOM  to  SWAN  further  improved  the  TC  wave  field. 

•  GEOS-5:  The  NASA  GEOS-5  atmosphere-ocean  general  circulation  model  is  designed  to 
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399  simulate  climate  variability  on  a  wide  range  of  time  scales,  from  synoptic  time  scales  to 

400  multi-century  climate  change.  Projects  underway  with  the  GEOS-5  AOGCM  include 

401  weakly  coupled  ocean-atmosphere  data  assimilation,  seasonal  climate  predictions  and 

402  decadal  climate  prediction  tests  within  the  framework  of  Coupled  Model  Intercomparison 

403  Project  Phase  5  (CMIP5;  Taylor  et  al.  2012).  The  decadal  climate  prediction  experiments 

404  are  being  initialized  using  the  weakly  coupled  atmosphere-ocean  data  assimilation  based 

405  on  MERRA  (Rienecker  et  al.  201 1).  All  components  are  coupled  together  using  ESMF 

406  interfaces. 

407  •  NEMS:  The  NEMS  modeling  system  under  construction  at  NOAA  is  intended  to 

408  streamline  development  and  create  new  knowledge  and  technology  transfer  paths  that 

409  bridge  the  NOAA  research  and  operational  centers  and  other  agency  efforts.  NEMS  will 

410  encompass  multiple  coupled  models,  including  future  implementations  of  the  Climate 

41 1  Forecast  System  (CFS;  Saha  2014),  the  Next  Generation  Global  Prediction  System 

412  (NGGPS;  Lapenta  2015),  and  regional  hurricane  forecast  models.  The  new  CFS  will 

413  couple  global  atmosphere,  ocean,  sea  ice  and  wave  components  through  the  NUOPC 

414  Layer  for  advanced  probabilistic  seasonal  and  monthly  forecasts.  NGGPS  is  being 

415  designed  to  improve  and  extend  weather  forecasts  to  30  days,  and  will  include  ocean  and 

416  other  components  coupled  to  an  atmosphere.  The  NEMS  hurricane  forecasting  capability 

417  will  have  nested  mesoscale  atmosphere  and  ocean  components  coupled  through  the 

418  NUOPC  Layer  for  advanced  probabilistic  tropical  storm  track  and  intensity  prediction. 

419  Early  model  outputs  from  the  atmosphere  (GSM),  ocean  (MOM5),  and  sea  ice  (CICE) 

420  three-way  coupled  system  in  NEMS  are  currently  being  evaluated. 

421  •  CESM:  The  CESM  coupled  global  climate  model  enables  state-of-the  art  simulations  of 


19 


422  Earth’s  past,  present  and  future  climate  states  and  is  one  of  the  primary  climate  models 

423  used  for  national  and  international  assessments.  A  recent  effort  involves  coupling 

424  HYCOM  to  CESM  components  using  NUOPC  Layer  interfaces.  A  scientific  goal  of  the 

425  HYCOM-CESM  coupling  is  to  assess  the  impact  of  hybrid  versus  depth  coordinates  in 

426  the  representation  of  our  present-day  climate  and  climate  variability.  The  project 

427  leverages  an  effort  to  couple  HYCOM  to  an  earlier  version  of  CESM,  CCSM3  (Lu  et  al. 

428  2013;  Michael  et  al.  2013). 

429  •  Grid  Remapping:  An  ongoing  collaboration  between  CESM  and  ESMF  led  to  joint 

430  development  of  the  parallel  ESMF  grid  remapping  tools.  These  are  now  widely  used  by 

431  modeling  groups  and  visualization  and  analysis  packages  including  NCL  and  UV-CDAT, 

432  and  have  enabled  projects  like  CESM  to  meet  critical  milestones  and  opened  doors  to 

433  new  research  initiatives.  For  example,  leveraging  ESMF  grid  remapping,  CESM  was  able 

434  to  create  offline  utilities  that  permit  researchers  to  run  CESM  on  user-defined  grids, 

435  including  regionally  refined  grids.4  ESMF  offline  remapping  has  also  enabled  the 

436  incorporation  of  the  Model  for  Prediction  Across  Scales  (MPAS)  ocean  model  as  a  new 

437  CESM  component.  Recent  efforts  are  focusing  on  migrating  the  off-line  grid  remapping 

438  into  a  run  time  capability  in  order  that  more  dynamic  and  adaptive  grids  can  be 

439  supported. 

440  ESPS  Opportunities  for  Managed  and  Interactive  Ensembles 

441  In  the  weather  and  climate  prediction  communities  ensemble  simulations  are  used  to  separate 

442  signal  from  noise,  reduce  some  of  the  model-induced  errors  and  improve  forecast  skill. 

443  Uncertainty  and  errors  come  from  several  sources: 

4  These  utilities  have  been  folded  into  the  publically  released  version  of  the  model  as  of  CESM  1.2.0. 
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(i)  Initial  condition  uncertainty  associated  with  errors  in  our  observing  systems  or  in  how 
the  observational  estimates  are  used  to  initialize  prediction  systems  (model 
uncertainty /errors  play  a  significant  role  here); 

(ii)  Uncertainty  or  errors  in  the  observed  and  modeled  external  forcing.  This  can  be  either 
natural  (changes  in  solar  radiation  reaching  the  top  of  the  atmosphere,  changes  in 
atmospheric  composition  due  to  natural  forcing  such  as  volcanic  explosions,  changes 
in  the  shape  and  topography  of  continents  or  ocean  basins),  or  anthropogenic 
(changes  is  the  atmospheric  composition  and  land  surface  properties  due  to  human 
influences); 

(iii)  Uncertainties  or  errors  in  the  formulation  of  the  models  used  to  make  the  predictions 
and  to  assimilate  the  observations.  These  uncertainties  and  errors  are  associated  with 
a  discrete  representation  of  the  climate  system  and  the  parameterization  of  sub-grid 
physical  processes.  The  modeling  infrastructure  development  described  here  is  ideally 
suited  to  quantify  uncertainty  due  to  errors  in  model  formulation,  and  where  possible 
reduce  this  uncertainty. 

To  account  for  initial  condition  uncertainty  it  is  standard  practice  to  perform  a  large  ensemble  of 
simulations  with  a  single  model  by  perturbing  the  initial  conditions.  The  ensemble  mean  or 
average  is  typically  thought  of  as  an  estimate  of  the  signal  and  the  ensemble  spread  or  even  the 
entire  distribution  is  used  to  quantify  the  uncertainty  (or  noise)  due  to  errors  in  the  initial 
conditions.  In  terms  of  uncertainty  in  external  forcing,  the  model  simulations  that  are  used  to 
inform  the  Intergovernmental  Panel  on  Climate  Change  (IPCC)  use  a  number  of  different 
scenarios  for  projected  greenhouse  gas  forcing  to  bracket  possible  future  changes  in  the  climate. 
In  both  of  the  examples  above,  it  is  also  standard  practice  to  use  multiple  models  to  quantify 
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467  uncertainty  in  model  formulation  and  to  reduce  model- induced  errors. 

468  The  use  of  multi-model  ensembles  falls  into  two  general  categories  both  of  which  are  easily 

469  accommodated  by  ESPS.  The  first  category  is  an  a  posteriori  approach  where  ensemble 

470  predictions  from  different  models  are  combined,  after  the  simulation  or  prediction  has  been  run, 

47 1  into  a  multi-model  average  or  probability  distribution  that  takes  advantage  of  complementary 

472  skill  and  errors.  This  approach  is  the  basis  of  several  international  collaborative  prediction 

473  research  efforts  (e.g.,  National  Multi-Model  Ensemble,  ENSEMBLES),  climate  change 

474  projection  (CMIP)  efforts,  and  there  are  numerous  examples  of  how  this  multi-model  approach 

475  yields  superior  results  compared  to  any  single  model  (e.g.,  Kirtman  et  al.  2013).  In  this  case,  the 

476  multi-model  average  estimates  the  signal  that  is  robust  across  different  model  formulations  and 

477  initial  condition  perturbations.  The  distribution  of  model  states  is  used  to  quantify  uncertainty 

478  due  to  model  formulation  and  initial  condition  errors.  While  this  approach  has  proven  to  be  quite 

479  effective,  it  is  generally  ad  hoc  in  the  sense  that  the  chosen  models  are  simply  those  that  are 

480  readily  available.  The  ESPS  development  described  here  allows  for  a  more  systematic  approach 

481  in  that  individual  component  models  (e.g.,  exchanging  atmospheric  components  CAM5  for 

482  GEOS-5)  can  easily  be  interchanged  within  the  context  of  the  same  coupling  infrastructure  thus 

483  making  it  possible  to  isolate  how  the  individual  component  models  contribute  to  uncertainty  and 

484  complementary  skill  and  errors.  For  simplicity  we  refer  to  the  interchanging  or  exchanging 

485  component  models  as  managed  ensembles. 

486  The  second  category  can  be  viewed  as  an  a  priori  technique  in  the  sense  that  the  model 

487  uncertainty  is  “modeled”  as  the  model  evolves.  This  approach  recognizes  that  the  dynamic  and 

488  thermodynamic  equations  have  irreducible  uncertainty  and  that  this  uncertainty  should  be 

489  included  as  the  model  evolves.  This  argument  is  the  scientific  underpinning  for  the  multi-model 
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490  interactive  ensemble  approach.  The  basic  idea  is  to  take  advantage  of  the  fact  that  the  multi- 

491  model  approach  can  reduce  some  of  the  model-induced  error,  but  with  the  difference  being  that 

492  this  is  incorporated  as  the  coupled  system  evolves.  In  ESPS  we  can  use  the  atmospheric 

493  component  model  from  say  CAM5  and  GEOS-5  simultaneously  as  the  coupled  system  evolves, 

494  and  for  example,  combine  the  fluxes  (mean  or  weighted  average)  from  the  two  atmospheric 

495  models  to  communicate  with  the  single  ocean  component  model.  Moreover,  it  is  even  possible  to 

496  sample  the  atmospheric  fluxes  in  order  to  introduce  state  dependent  and  non-local  stochasticity 

497  into  the  coupled  system  to  model  the  uncertainty  due  to  model  formulation.  Forerunners  of  the 

498  approach  have  been  implemented  within  the  context  of  CCSM  to  study  how  atmospheric  weather 

499  noise  impacts  climate  variability  (Kirtman  et  al.  2009,  Kirtman  et  al.  2011)  and  seasonal 

500  forecasts  in  the  NOAA  operational  prediction  system  (Stan  and  Kirtman  2008). 

501  FUTURE  DIRECTIONS 

502  Next  steps  include  continued  development  of  NUOPC -based  modeling  systems,  ongoing 

503  improvements  to  ESPS  metadata  and  user  access  information,  exploration  of  the  opportunities 

504  ESPS  affords  in  creating  new  ensemble  systems,  and  addition  of  capabilities  to  the  infrastructure 

505  software  itself.  Whether  to  extend  the  ESPS  to  other  types  of  components  is  an  open  question. 

506  Developers  have  already  implemented  NUOPC  Layer  interfaces  on  components  that  do  not  fall 

507  into  the  initial  ESPS  model  categories,  including  the  WRF-Hydro  hydrology  model,  the 

508  Community  Land  Model  (CLM),  and  the  Ionosphere-Plasmasphere  Electrodynamics  (IPE) 

509  model. 

510  The  continued  incorporation  of  additional  processes  into  models,  the  desire  for  more  seamless 

511  prediction  across  temporal  scales,  and  the  demand  for  more  information  about  the  local  impacts 

512  of  climate  change  are  some  of  the  motivations  for  linking  frameworks  from  multiple  disciplines. 
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513  The  NSF-funded  Earth  System  Bridge  project  is  building  converters  that  will  enable  NUOPC 

514  codes  to  be  run  within  the  Community  Surface  Dynamics  Modeling  System  (CSDMS),  which 

515  contains  many  smaller  models  representing  local  surface  processes,  and  CSDMS  codes  to  be  run 

516  within  ESMF.  The  ESMF  infrastructure  is  also  being  used  to  develop  web  service  coupling 

517  approaches  in  order  to  link  weather  and  climate  models  to  frameworks  that  deliver  local  and 

518  regional  information  products  (Goodall  et  al.  2013). 

519  A  critical  aspect  of  future  work  is  the  evaluation  and  evolution  of  NUOPC  and  ESMF  software 

520  for  emerging  computing  architectures.  A  primary  goal  is  for  common  infrastructure  such  as  the 

521  NUOPC  Layer  to  do  no  harm,  and  allow  for  optimizations  within  component  models.  However, 

522  NUOPC  infrastructure  also  offers  new  optimization  opportunities  for  coupled  systems.  The 

523  formalization  of  initialize  and  run  phases,  which  allows  components  to  negotiate  with  each  other 

524  for  resources,  holds  great  potential  in  dealing  with  systems  that  have  an  increasing  number  of 

525  components,  and  will  need  to  run  efficiently  on  accelerator-based  compute  hardware.  Among  the 

526  planned  extensions  to  NUOPC  protocols  are  hardware  resource  management  between 

527  components  and  the  negotiation  of  data  placement  of  distributed  objects.  Both  extensions 

528  leverage  the  ESMF  “virtual  machine”  or  hardware  interface  layer,  already  extended  under  the 

529  ESPC  initiative  to  be  co-processor  aware.  The  awareness  of  data  location  can  also  be  used  to 

530  minimize  data  movement  and  reference  data  where  possible  during  coupling.  Finally,  there  is 

531  interest  in  optimizing  the  grid  remapping  operation  between  component  grids  in  the  mediator  by 

532  choosing  an  optimal  decomposition  of  the  transferred  model  grid.  This  optimization  requires 

533  extra  negotiation  between  the  components  which  could  be  made  part  of  the  existing  NUOPC 

534  component  interactions. 

535  CONCLUSION 
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536  Through  the  actions  of  a  succession  of  infrastructure  projects  in  the  Earth  sciences  over  the  last 

537  two  decades,  a  common  model  architecture  (CMA)  has  emerged  in  the  U.S.  modeling 

538  community.  This  has  enabled  high-level  model  components  to  be  wrapped  in  community- 

539  developed  ESMF  and  NUOPC  interfaces  with  few  changes  to  the  model  code  inside,  in  a  way 

540  that  retains  much  of  the  native  model  infrastructure.  The  components  in  the  resulting  systems 

541  possess  a  well-defined  measure  of  technical  interoperability.  The  ESPS,  a  collection  of  multi- 

542  agency  coupled  weather  and  climate  systems  that  complies  with  these  standard  interfaces,  is  a 

543  tangible  outcome  of  this  coordination.  It  is  a  direct  response  to  the  recommendations  of  a  series 

544  of  National  Research  Council  and  other  reports  recommending  common  modeling  infrastructure, 

545  and  a  national  asset  resulting  from  commitment  of  the  agencies  involved  in  Earth  system 

546  modeling  to  work  together  to  address  global  challenges. 
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569  SIDEBAR  I: 

570  LINKED  AND  LEVERAGED: 

571  THE  EVOLUTION  OF  COUPLED  MODEL  INFRASTRUCTURE 

572  First  generation  ( 1 996-200 1 )  Model  coupling  technologies  were  initially  targeted  for 

573  specific  modeling  systems,  often  within  a  single  organization.  Infrastructure  that  arose  out  of 

574  model  development  during  this  period  included  the  Flexible  Modeling  System  (FMS)  at  the 

575  Geophysical  Fluid  Dynamics  Faboratory,  the  Goddard  Earth  Modeling  System  (GEMS;  NASA 

576  GSFC  1997),  and  the  Climate  System  Model  (CSM;  Boville  and  Gent  1998)  and  Parallel 

577  Climate  Model  (PCM;  Washington  et  al.  2000)  flux  couplers  at  NCAR.  Each  of  these  systems 

578  coordinated  functions  such  as  timekeeping  and  I/O  across  model  components  contributed  by 

579  domain  specialists,  and  implemented  component  interfaces  for  field  transformations  and 

580  exchanges. 
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581  Second  generation  (2002-2006)  Recognizing  similar  functions  and  strategies  across  first 

582  generation  model  infrastructures,  a  multi-agency  group  formed  a  consortium  to  jointly  develop 

583  an  Earth  System  Modeling  Framework  (ESMF).  ESMF  was  intended  to  limit  redundant  code 

584  and  enable  components  to  be  exchanged  between  modeling  centers.  Also  at  this  time,  within 

585  DOE,  the  Common  Component  Architecture  (CCA;  Bernholdt  et  al.  2006)  consortium 

586  introduced  a  more  precise  definition  of  components  into  the  high  performance  computing 

587  community,  and  members  of  the  Model  Coupling  Toolkit  (MCT)  project  worked  with  CSM 

588  (now  CCSM  -  the  Community  CSM)  to  abstract  low-level  coupling  functions  into  the  MCT 

589  general-purpose  library  and  develop  a  new  CCSM  coupler  (CPL7). 

590  Third  generation  (2007-20 1 4)  A  third  generation  of  development  began  as  multi-agency 

591  infrastructures  began  to  mature  and  refactor  code,  assess  their  successes  and  deficiencies,  and 

592  encounter  new  scientific  and  computational  challenges.  Both  NASA,  with  the  Modeling  Analysis 

593  and  Prediction  Layer  (MAPL;  Suarez  et  al.  2007)  and  the  National  Unified  Operational 

594  Prediction  Capability  (NUOPC),  a  group  of  NOAA,  Navy  and  Air  Force  operational  weather 

595  prediction  centers  and  their  research  partners,  added  conventions  to  ESMF  to  increase 

596  component  interoperability.  Similar  refactoring  efforts  took  place  in  other  communities  such  as 

597  surface  dynamics  (Peckham  et  al.  2013)  and  agriculture  (David  et  al.  2010).  The  demands  of 

598  high  resolution  modeling  and  the  advent  of  unstructured  grids  pushed  ESMF  to  develop  new 

599  capabilities  and  products,  and  MCT  and  CCSM  -  now  CESM  -  to  introduce  new  communication 

600  options.  In  this  wave  of  development,  the  capabilities  of  shared  infrastructure  began  to  equal  or 

601  outperform  those  developed  by  individual  organizations. 

602  What  next?  (20 1  5  -  )  Although  some  infrastructure  projects  have  disappeared  or  merged, 

603  projects  from  all  three  generations  of  development  are  still  in  use,  and  increasingly  their 
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interfaces  may  coexist  in  the  same  modeling  system.  Future  development  is  likely  to  include 
more  cross-disciplinary  projects  like  the  Earth  System  Bridge  (see  Peckham  et  al.  2014),  which 
is  defining  a  formal  characterization  of  framework  elements  and  behaviors  (an  Earth  System 
Framework  Description  Language,  or  ES-FDL),  and  using  it  to  explore  how  to  link  components 
that  come  from  different  communities  that  have  their  own  infrastructures  (e.g.  climate, 
hydrology,  ecosystem  modeling). 

SIDEBAR  II 

LIMITS  OF  COMPONENT  Possible  image  for  Sidebar  II. 

INTEROPERABILITY 

NUOPC  Layer  compliance  guarantees  certain 
aspects  of  technical  interoperability,  but  it  does  not 
guarantee  that  all  components  of  the  same  type,  for 
instance  all  NUOPC-wrapped  atmosphere  models, 
will  be  scientifically  viable  in  a  given  coupled 
modeling  system.  A  simple  example  of  scientific 
incompatibility  is  one  in  which  the  exported  fields 
available  do  not  match  the  imported  fields  needed  for  a  component  to  run.  Other 
incompatibilities  can  originate  in  how  the  scope  of  the  component  is  defined  (i.e.,  which  physical 
processes  are  included),  and  in  assumptions  about  how  the  component  will  interact  with  other 
components. 5  For  example,  some  modeling  systems  implement  an  implicit  interaction  between 

5  Alexander  and  Easterbrook  2011.  provide  a  high-level  look  at  variations  in  the  component  architecture  of  climate 
models. 
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atmosphere  and  land  models  while  others  take  a  simpler  explicit  approach.  Whether  or  not  a 
component  can  adapt  to  a  range  of  configurations  and  architectures  is  determined  as  well  by 
whether  scientific  contingencies  are  built  into  it  by  the  developer.  The  components  in  the  ESPS 
are  limited  to  major  physical  domains  since  many  of  the  models  in  this  category,  such  as  CAM, 
CICE,  and  HYCOM  have  been  built  with  the  scientific  flexibility  needed  to  operate  in  multiple 
modeling  systems  and  coupling  configurations. 
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